Sample-Optimal Identity Testing with High Probability

We study the problem of testing identity against a given distribution (a.k.a. goodness-of-fit) with a focus on the high confidence regime. More precisely, given samples from an unknown distribution p over n elements, an explicitly given distribution q, and parameters 0 < ε, δ < 1, we wish to distinguish, with probability at least 1 − δ, whether the distributions are identical versus ε-far in total variation (or statistical) distance. Existing work has focused on the constant confidence regime, i.e., the case that δ = Ω(1), for which the sample complexity of identity testing is known to be Θ( √ n/ε). Typical applications of distribution property testing require small values of the confidence parameter δ (which correspond to small “p-values” in the statistical hypothesis testing terminology). Prior work achieved arbitrarily small values of δ via black-box amplification, which multiplies the required number of samples by Θ(log(1/δ)). We show that this upper bound is suboptimal for any δ = o(1), and give a new identity tester that achieves the optimal sample complexity. Our new upper and lower bounds show that the optimal sample complexity of identity testing is Θ ( 1 ε2 (√ n log(1/δ) + log(1/δ) )) for any n, ε, and δ. For the special case of uniformity testing, where the given distribution is the uniform distribution Un over the domain, our new tester is surprisingly simple: to test whether p = Un versus dTV (p, Un) ≥ ε, we simply threshold dTV (p̂, Un), where p̂ is the empirical probability distribution. We believe that our novel analysis techniques may be useful for other distribution testing problems as well. ∗Supported by NSF Award CCF-1652862 (CAREER) and a Sloan Research Fellowship. †Supported by the NSF under Grant No. 1420692. ‡Supported by the NSF Graduate Research Fellowship under Grant No. 1122374, and by the NSF under Grant No. 1065125.